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(54) Title: INTERACTIVE SURVEILLANCE SYSTEM 
(54) Bezeichnung: INTER AKTIVES tJBERWACHUNGSS YSTEM 
(57) Abstract 

A process is disclosed for observing or monitoring an area 
in space. Images of the area (1) to be observed/monitored arc 
taken by means of at least one camera (2), the converted image 
information is sent to a control computer (3), the control computer 
(3) checks the images for changes in the observed area (1), the main 
direction and/or speed of optically perceptible changes (movements) 
in the observed area are analysed by analysis of the optical flow, 
the analysis result is converted into control instructions sent to a 
controller (4) that mechanically moves the camera, and if required 
the position and/or size of the image section that may be taken by 
the camera is modified. Also disclosed is an interactive surveillance 
system with a control computer (3) connected to or integrated into a 
camera (2) or into one among several cameras. The control computer 

(3) receives and processes the data recorded by the camera(s) and 
sends control signals to a zoom lens (5). A mechanical controller 

(4) reacts to impulses from the control computer (3) and modifies 
the recording section of the area (1) to be observed or the size of the 
image section (5). A device reacts to signals from the control computer 
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(3) and opens a normally closed passage. 


(57) Zusammenfassung 


r,m J? fe B J ndung ^i^^ Y (a 5?" en Beobachten ^ Obenractai eines raumlichen Bereiches, umfassend das Aufnehmen von 
Bildem aus dem zu beobachtenden/uberwachenden Bereicli (1) mil Hilfe mindestens einer Kamera (2), das Scnden der umgewandelten 
Bil^nfomatioii an emen Steueirechner (3) das Auswerten derBilder im Steuerrechner (?) in bezug auf Veranderungen im zu uberwachenden 
Bereicn (1), das Analysieren der Hauptnchtung und/oder der Geschwindigkeit von optisch wahmehmbaren Veranderungen (Beweeuneen> 
im zu flberwachenden Bereich mil Hilfe der Analyse des optischen Flusses, das Umsetzen des Analyseergebnisses in Steuerbefehle 
zum mechamschen Bewegen der Kamera an erne Steuereinrichtung (4), und ggf. das Verandem des von der Kamera aufhehmbaren 
Bildausschnittes in bezug auf Lage und/oder GrdBe; des weitereh betrifft die Erfindung eta interaktives Uberwachungssystem Telches 
emen .nut einer Kamera (2) oder einer von mehreren Kameras (2) verbundenen Oder darin integrierten Steuerrechner (3), der die von 
dei/den Kamera(s) aufgenommenen Dater . aufnimmt und verarbeitet und Steuersignale an ein Zoom-Objektiv (5) senden kann femer eine 
mechawsche ; Steuereimichhmg (4) die auf Impulse des Steuerrechners (3) bin den Aufnahmeausschhitt aus dem zu ObemaSe^S 
(1) Oder die GroBedes Bildausschnittes (5) verandem kann, sowie eine Einrichtung zum Offnen einer Ublicherweise verschlossene^ 
aufweist, die auf Signale des Steuerrechners (3) reagiert. "-"ciwcise veiscniossenen Passage 
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Xnteraktives TJfeerwachungs system 

Die vorliegende Erfindung betrifft ein interaktives 
Beobachtungs- oder Uberwachurigs system sowie ein Verfahren zum 
5 Uberwachen bzw. Beobachten eines raumlichen Bereiehes, 

insbesondere zur Personenuberwachung oder Identif izierung von 
Personen. 

Der Stand der Technik kexmt Kamerasysteme fur 
10 Uberwachungsaufgaben. Diese Kamerasysteme beschranken sich 

jedoch auf reine Visualisierungs- und Ubertragungsaufgaben,. d.h. 
die Bildinf ormationen aus dem zu uberwachenden Bereich werden 
zur direkten, gleichzeitigen Beobachtung an Monitore in einer 
Zentrale ubermittelt, oder sie werden auf Videoband 
15 abgespeichert , um spater abgerufen werden zu konnen, bzw. sie 
werden nach einiger Zeit automatisch geloscht . 

Aus der EP 90 104 469.3 ist ein Verfahren zur Detektion von 
Intrudern bekanrit, in welchem mit einer Videokamera und einer 
20 Video-Bild-Verarbeitungseinheit ein vorgegebener 

Uberwachungsbereich 'Qberwacht wird. Dabei werden die zu 
detektierenden Objekte durch zwei zeitlich versetzte Videobilder 
mit Hilf e eines Dif f erenzbildverf ahrens erkannt . 

25 Prinzipiell werden im Dif f erenzbildverf ahren 

auf einanderf olgende Bilder voneinander abgezogen. Tritt eine 
hinreichend groSe Differenz zwischen zwei Bildern auf # so wird 
diese Differenz als "Bewegung" identif iziert bzw. analysiert. 

30 Der Einsatz solcher Verfahren hat jedoch Nachteile. So wird die 
Gesamtbewegungsrichtung im Falle komplexer Strukturen und einer 
inhomogenen Bewegungen (Beispiel: ein Arm schlenkert gegen die 
Lauf richtung) haufig inkorrekt ermittelt, da die 
Bewegungsrichtung anhand des ermittelten Schwerpunktes des zu 

35 beobachtenden Objekts analysiert wird. 
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Aufgabe der vorliegenden Erfindung ist es, ein 
Uberwachungssystem bereitzustellen, in welchem die Kamera in 
Abhangigkeit vom Bildinhalt automatisch gesteuert und 
nachgeffihrt werden kann, wobei die Gesamtbewegungsrichtung mit 
hoher Genauigkeit ermittelt wird. 

Gelost wird die vorgenannte Aufgabe mit Hilfe des Verfahrens 
gemaS Anspruch 1 . 

Das interaktive Uberwachungssystem ist eine "intelligente 
Kamera" / die selbsttatig und schnell auf durch eine Analyse des 
optischen Flusses gewonnene Bildsignale reagieren kann. Es 
handelt sich dabei urn eine Echtzeit-Bildauswe.rtung einer oder 
mehrerer Kameras, die der Bewegung in einem Uberwachungsbereich 
nachgefuhrt werden kann bzw. k6nnen und/oder fiber die 
Veranderung des Bildausschnittes (Zoom) eine VergroEerung 
interessierender Bereiche bewirkt und somit z.B. die Gesichter 
von Menschen herausvergroSern kann. Das System eignet sich fur 
die Uberwachung von Gebauden (Eingangen, Tiefgaragen) unddgl. 
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Figur la-c zeigt die Auswertung von zu. verschiedenen Zeiten (t 
urxd t+1) aufgenommenen Bildern (Fig. la) und zwar mit Hilfe der 
Analyse des optischen Flusses (Fig. lb) bzw. mit Hilfe eines 
Dif ferenzbildverfahrens (Fig. lc) . In Figur lb erkennt man ein 
optisches FluSvektorf eld . Das Kreuz zeigt jeweils den Ursprung 
der Bewegungsrichtung, Lange und Richtung des jeweiligen 
Striches weisen Lange und Richtung des Bewegungsvektors . 

Durch z . B . Vektorsummation aller Einzelrichtungen kann nun auf 
eine Gesamtbewegungsrichtung von Objekten im Bild geschlossen 
und somit die Kamera nachgefuhrt werden. Mit einem reinen 
Differenzbild ist dies, wie aus Figur lc erkennbar, nicht 
moglich. 

Figur 2 zeigt ein System mit einer Autofocus- Kamera 2 mit einem 
Zoom-Objektiv 5, die. auf einen Uberwachungsbereich 1 gerichtet; 
ist. Die Kamera gibt die aufgenommenen Bildsignale an einen 
Steuerrechner 3 weiter, der wiederum mit einer mechanischen y 
Steuereinrichtung 4 verbunden ist. Die Kamera kann in 
verschiedener Weise einer ggf . detektierteri Bewegung nachgehen, 
beispielsweise indem sie die Beweguiigsrichtung analysiert und 
dieser Richtung im zu uberwachenden Bereich folgt oder indem sie 
eine VergroSerung eines interessierenden Bildausschnittes durch 
Zoomen bewirkt . 

Die Kamera kann kontinuierlich Bilder vom zu uberwachenden 
Bereich an den Steuerrechner senden* SelbstverstSndlich sind 
auch intermittierende Aufnahmen oder ein diskontinuierliches 
Aufnehmen (nur zu interessierenden, vorherbestimmten oder nicht 
vorherbestimmten Zeitabschnitten) moglich. Im Anf angszustand ist 
dabei das an der Kamera angeschlossene Zoom-Objektiv auf den 
grofitmoglichen Blickwinkel eingestellt. 

Die Bilder werden vom Steuerrechner (3) kontinuierlich 
ausgewertet, wobei nach Bewegung im Bild gesucht wird. 
Tritt Bewegung im Bild auf, so wird mit Hilfe einer Analyse des 
optischen Flusses die Hauptrichtung der Bewegung und deren 
Geschwindigkeit ermittelt. Durch die Vektorsummation aller 
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Einzelrichtungen wird dabei auf eine Gesamtbewegungsrichtung von 
Objekten im Bild geschlossen, worauf die Kamera geeignet 
nachgefuhrt werden kann, so dafi das bewegte Objekt immer voir im 
Bild ist. 

Der Steuerrechner ermittelt aus dem Ort der Bewegung im Bild 
Steuerbefehle, urn die Kamera auf eine geeighete VergroSerung der 
Bewegung nachzustellen. Dabei werden die Bewegungen bezuglich 
des "Blickwinkels" der Kamera, also ihrer Lage bzw. des 
Ausschnittes aus dem zu uberwachenden Bereich, durch eine 
mechanische Steuereinrichtung bewirkt, die ihre Impulse vom 
Steuerrechner erhalt. Die Impulse zum Verandern des Zoom- 
Ausschnittes der Kamera werden vom Steuerrechner an die Kamera 
zurvickgeleitet . Durch die Kombination dieser beiden 
"intelligenten" Mafinahmen wird die Ursache der Bewegung (die 
Person, das Tier, das sich bewegt) optimal ins Bild gebracht und 
vergrofiert, so daS z.B. die Bewegungen einer Person vollstandig 
von der Kamera aufgenommen werden, solange sie sich im zu 
uberwachenden Bereich auf halt . 

Parallel dazu kann mit einera Mustervergleichsverf ahren, wie es 
z.B. aus dem Stand der Technik bekannt ist, im Bild nach 
gesichtsfarbigen Objekten gesucht werden und speziell ein 
Gesicht einer Person im Oberwachungsbereich herausvergrofiert 
werden. So kann ein derartiges Mustervergleichsverf ahren z.B. 
nach ellipsenformigen Objekten bestimmter Ausrichtung, GroSe und 
Helligkeit suchen, die im Normalfall Gesichter darstellen. 

Vorzugsweise schaltet die Kamera in regelmaJSigen Abstanden auf 
den giinstigsten Blickwinkel und den ihr grogtmoglichen 
Ausschnitt zuriick, um evtl . weitere Bewegungen im 
Uberwachungsbereich zu detektieren. 

Die interactive Kamera kann auch mit existierenden Gesicht s- und 
Personenerkennungssystemen gekoppelt werden, um Aussagen iiber 
die Identitat der im Uberwachungsbereich befiridlichen Person zu 
ermitteln. Hierzu wird z.B. ein Ausschnitt, der anhand der 
Bewegungsanalyse als zu einem bewegten Gesicht gehorig erkannt 
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wurde, herausvergrofiert und dieses Teilbild einem Muster- 
vergleich zugefiihrt, der in einer Datenbank mit abgespeicherten 
Gesichtsbildern nach dem Namen der entsprechenden Person sucht . 

In einer besonderen Ausgestaltung des erf indungsgemafien 
Verfahrens kann durch automat ischen Miistervergleich des 
aufgenommenen Bewegungsmusters mit abgespeicherten Mustern 
zwischen Personen und anderen bewegten Objekten (Blatter im 
Wind, Tieren usw.) unterschieden werden, urn Fehlalarme zu 
vermeiden. 

Selbstverstandlich kann das interaktive Uberwachungssystem nicht 
nur mit einer Kamera, wie oben beschrieben, ausgestattet sein. - 
Bei der Verwendung mehrerer Kameras konnen 

Stereobildauswertungsverf ahren zur Detektion und Erkennung von 
bewegten Objekten herangezogen werden. 

Das vorstehend beschriebene interaktive System zur Beobachtung < 
bzw. Uberwachung von Bereichen wie Gebauden (Eingangen, 
Tiefgaragen) oder dgl. fuhrt durch eine Echtzeit-Bildauswertung 
die Kamera oder die Kameras der Bewegung nach und vergroSert den 
bewegten Gegenst and optimal heraus. Der bewegte Gegenst and, z.B. 
eine menschliche Person, kann automatisch jsy- 
klassif iziert/identif iziert werden, wobei sich Fehlalarme durch 
die Bewegung von Tieren, Blattern usw. vermeiden lassen. In der 
Kombination mit bekannten Bild- und Signalverarbeitungsverf ahren 
eigne t sich das Verf ahren und das System der vorliegenden 
Erfindung zur namentlichen .Identif ikation der im 
Uberwachungsbereich bef indlichen Personen und dadurch auch als 
Eingangskontroll system fur Bereiche, die nur einer definierten 
Personengruppe zug&nglich sein diirfen (Anspruche 8 und 12). 
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Anspruche: 

1. Verfahren zum Beobachten oder Uberwachen eines raumlichen 
Bereiches, umfassend 

das Aufnehmen von Bildern aus deih zu 
beobachtenden/uberwachenden Bereich (1) mit Hilfe 
mindestens einer Kamera (2) , 

das Senden (2a) der umgewandelten Bildinf ormation (A/D) 
an einen Steuerrechner (3) , 

das Auswerten der Bilder im Steuerrechner (3) , um eine 
erste Veranderung im zu uberwachenden Bereich (1) zu 
erf assen, : 

das vektorielle Ermitteln der Bewegungsrichtung der 
optisch wahrgenommenen ersten Veranderung (Bewegungen) 
im zu uberwachenden Bereich mit Hilfe des optischen 
Flusses, 

das Umsetzen des Ermittlungsergebnisses in 
Steuerbefehle an eine Steuereinrichtung (4) zum 
mechanischen Bewegen der Kamera, und ggf. 
das Verandern (5) des von der Kamera erfaSten 
Bildausschnittes in Bezug auf Grofie. 

2. Verfahren nach Anspruch 1, worin beim Erkennen von Bewegung 
im zu uberwachenden Bereich optische, akustische oder 
elektrische (Alarm-) Signale abgegeben werden. 

3. Verfahren nach einem der obigen Anspruche, worin der 
Bildausschnitt mindestens einer Kamera (2) beim Erkennen von 
Bewegung durch Verandern eines Zoom-Ob jektes (5) vergroSert 
wird. 

4. Verfahren nach einem der voranstehenden Anspruche, wobei 
zusatzlich parallel mit einem Mustervergleichsverf ahren im 
raumlichen Bereich nach gesichtsf ormigen Objekten gesucht 
wird und bei Erkennen des sich bewegenden Objektes als 
Gesicht dieses mit Hilfe eines Zooms (5) der Kamera (2) 
vergroSert und/oder seiner Bewegung mit der Kamera (2) 
nachgefuhrt wird. 
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5. Verfahren nach einem der voranstehenden Aiispriiche, worin in 
vorbestimmten AbstSnden die ursprungliche Lage und/oder der 
ursprungliche Bildausschnitt der Kamera (2) 
wiederhergestellt werden, urn eine zweite oder dritte 
Veranderung im raumlichen Bereich (ebenf alls) zu erfassen. 

6. Verfahren nach einem der obigen Ansprxiche , worin das Bilder- 
Aufnehmen kontinuierlich oder intermittierend ist. 

7. Verfahren nach einem der voranstehenden Ansprxiche, wobei dem 
Steuerrechner (3) zusStzlich Inf ormationen uber Gesichter 
oder Personen aus einem Personenidentif ikationsverf ahren 

• zuganglich sind, 

8. Verfahren nach Anspruch 7, worin die Bildinf ormationen der 
Kamera (2) mit gespeicherten Inf ormationen zu definierten 
Personen verglichen wird und der Steuerrechner (3) mit einer 
Einrichtung verbunden ist, mit deren Hilfe beim Feststellen ' 
des Vorhandenseins von ausreichender Ubereinstimmung 4 
dahingehend, dafi Personenidentitat angenommen wird, eine 
liblicherweise verschlossene Passage geoffnet werden kann- 

9. Verfahren nach einem der voranstehenden Arispruche, wobei * r 
analysierte Bewegungsablauf e mit den Signalen bereits 
gespeicherter Bewegungsablaufe verglichen werden und die 
Abgabe von (Alarm-) Signalen nur dann erfolgt, wenn der 
Vergleich deutliche Abweichungen ergibt . 

10. Verfahren nach Anspruch 9, wobei der Bewegungsvergleich 
kontinuierlich oder intermittierend erfolgt. 

11. Verfahren nach einem der voranstehenden Anspriiche, worin die 
auszuwertenden Bilder Stereobilder sind, die von 
insbesondere zwei Kameras gleichzeitig aufgenommen werden, 
urn der Haupt-Bewegungsrichtung (vl,v2) der ersten und der 
zv/eiten "Veranderung" jeweils eine erste (vl , ,v2 l ) und eine 
zweite (yl",v2") - senkrecht zueinander orientierte - 
Bewegeungskomponente zuzuweisen. 
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12. Verfahren nach einem der voranstehenden Anspruche, worin 
nach zyklischem Wiederherstellen des ursprunglichen 
(groStmoglichen) Bilsausschnitts einer der zwei Kameras (2), 
die ira Bildausschhitt zuruckgestellte Kamera eine 
"Veranderung" als solches ermittelt und die nicht 
zuruckgestellte Kamera der Bewegung der schon erfafiten 
"Veranderung" weiter nachgefiihrt wird. 

13. Verfahren nach einem der voranstehenden Anspruche, worin die 
zyklische im Bildausschnitt zuruckgestellte Kamera nach 
vorbestimmter Zeit der Nicht -Erf assung einer weiteren 
"Veranderung" auf die von der anderen Kamera bereits erfaSte 
(erste) "Veranderung" schnell zielgefiihrt wird, urn mit 
beiden Kameras mehrere Bewegungskomponenten (vl,v2; vl',v2'; 
vl» ,v2") oder ein Stereobild zu erfassen. 

14. Interaktives Uberwachungssystem, umfassend 
mindestens eine Kamera (2) mit einem Zoom-Ob jektiv (5), 
einen mit einer Kamera oder einer der Kameras verbundenen 
oder darin integrierten Steuerrechner (3) , der die von 
der/den Kamera (s) auf genommenen Daten aufnimmt und 
verarbeitet und Steuersignale an das Zoom-Ob jektiv sendet, 
eine mechanische Steuereinrichtung (4), die auf Impulse des 
Steuerrechners (3) hin den Aufnahmeausschnitt aus dem zu 
uberwachenden Bereich (1) oder die Grofie des 
Bildausschnittes (5) verandem kann, sowie 

eine Einrichtung zum Offnen einer verschlossenen Passage, 
die auf Signale des Steuerrechners (3) reagiert. 

15. Uberwachungssystem nach Anspruch 12, zusatzlich umfassend 
eine Einrichtung zum Erzeugen und Senden von akustischen, 
optischen und/oder elektrischen (Alarm-) Signalen. 

16. Uberwachungssystem nach Anspruch 14 oder 15, bei dem zwei 
Kameras (2) mit ihrem Ruheachse (100) etwa rechtwinklig zu 
einem Zentrum des raumlichen Bereichs (1) angeordnet sind. 
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Bildt 


Figur 1a 


Bildt+1 



Figur 1b 
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(54) Method and apparatus for sensing object located within visual field of Imaging device 


(57) An apparatus and a method is provided for 
sensing an object within an imaging field of an imaging 
device (121). The data of the reference background pic- 
ture taken within the imaging field is stored in a storage 
unit (124). The data of the picture taken by the imaging 
device (121) is read. The data of the reference back- 
ground picture is compared with the read picture data at 
each pixel so that the difference value between both of 


the data may be yielded. The difference value is com- 
pared with a predetermined threshold value at each 
pixel. Based on the compared result, binarization data 
about an image of the object is produced. Based on the 
binarization data, the object is sensed. Then, the data of 
the reference background picture stored in the storage 
unit (1 24) is properly updated with the new data. 
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Description 

The present invention relates to a system for monitoring a picture which system uses an imaging device such as a 
television camera. More particularly, the invention relates to a method and an apparatus for sensing an object in the sys- 
s tern for monitoring a picture which method and apparatus are preferable in the case of needing so high reliability such 
as the case of sensing an object coming into an dangerous zone. 

In recent days, a picture monitoring system arranged to use a TV camera has been made public. Of such monitor- 
ing systems, there has been requested a system for automatically sensing an object such as a person or a car coming 
into a field of a monitor based on a video signal independently of human eyes and issuing a predetermined report and 
10 alarm. This type of automatic monitoring system is required to sense an object by processing a video signal produced 
by a TV camera. 

As one of the systems for sensing an object by processing such a video signal, there has been known and widely 
used as a difference method wherein the latest picture coming to the system in sequence is compared with the one pre- 
vious picture, a difference of a luminance for each corresponding pixels is obtained, and an area with a large difference 
15 as an object area. 

The basic flow of the system for sensing a difference value will be described with reference to the flowchart of Fig. 
8. As shown, at a step 81 , a current picture is captured from a TV camera. At the next step 82, a difference of a lumi- 
nance is obtained between a pre-stored background picture and the captured picture. Then, at a step 83, binarization 
process is made for the difference value. At last, at a step 84, the digital value is compared with a predetermined thresh- 
20 old value for sensing if an object exists based on the compared result. 

For example, as shown in Fig. 9A, assume that the background picture taken by a TV camera is a railroad crossing. 
The picture shown in the left hand of Fig. 9B is taken when one car 90 is coming into this crossing. The picture shown 
in the right hand of Fig. 9B is derived from the binarization of the difference value of the pixel data for each pixel between 
the background picture of Fig. 9A and the picture shown in the left hand of Fig. 9B. 
25 In this type of system, partial breaks or gaps 91 appear in an object to be sensed as indicated in the picture shown 
in the right hand of Fig. 9B. These breaks or gaps may bring about an error in sensing an object. 

However, this drawback may be overcome by applying a dilation and erosion operation to the system. 
As an applied example of a system for sensing a moving object by processings difference, for example, a writing 
titled as "Intelligent Image Handling with Image Recognition Technique" written by Ueda, et. a!., has been published in 
30 n O plus E" No. 176, pp. 122 to 136. 

The system for sensing a moving object, published in this writing, is arranged to sense an object on -three pictures 
continuing in time. As shown in Fig. 1 1 , at first, assume that the three continuous pictures 111, 112, 113 in time are 
input. Luminance differences between the pictures 1 1 1 and 112 and between the pictures 1 12 and 1 13 are calculated. 
These two luminance difference values are made binarized arid then are subject to the dilation and erosion operation, 
35 as a result of which pictures 1 1 4 and 1 1 5 are obtained. Next, these binary images (pictures) 1 1 4 and 1 1 5 are ANDed 
for deriving common portions between these pictures, on which portions a picture 116 of the object is obtained. 

In the aforementioned discJosed method, however, the following erroneous sensing may take place. That is, though 
an object is not actually coming into the view field, an object is erroneously sensed. For example, as shown in Fig. 9C, 
if the change of a weather or lighting conditions leads to change of brightness of an overall object to be imaged, an 
40 object 93 (railroad) may be erroneously sensed though no object is coming into the view field. As shown in Fig. 9D, if 
an object located out of the view field is made so luminous that a reflected ray 94 may take place in the area within an 
imaging field, the reflected ray is erroneously sensed as the object 95. Further, as shown in Fig. 9E, if the shadow 96 
of the object located out of the field is falling on the area within the imaging field, the shadow may be erroneously 
sensed as the object 97. 

45 By the way, this differential calculus needs a reference background picture (for example, Fig. 9A). Hence, the refer- 
ence background picture is created in advance and then pre-stored in a memory. To create the reference background 
picture, there has been known a system arranged to use a video signal (normally, a video signal of one frame) taken 
when no object is coming into the view field as the reference background picture according to the operator's selection 
or a system arranged to average a predetermined number of frames composing a picture at each pixel and to use the 

so averaged frames as the reference background picture. 

In the former system, the instantaneous picture is selected on the human determination. If any noise is mingled into 
the video signal by chance, the noise is held in the background picture. It is thus difficult to obtain the exact background 
picture. Further, from a view point of the noise circumstances in the transmission system, the mingling of the noise is 
not so rare. In the practical use, therefore, it is a great obstacle. 

55 In the latter system, to obtain the exact background picture, it is necessary to make the number of frames compos- 
ing the picture to be averaged as numerous as 1500 frames, for example. As a result, a large time gap (about 50 sec- 
onds in a frame frequency of 30 Hz) takes place between a time when a picture for creating the reference background 
picture is input and a time when a difference is processed for sensing the object. Hence, for example, if the imaging field 
is gradually made gloomier at dusk, for example, the time gap makes it impossible to create so exact a reference back- 
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ground picture as allowing the object to be sensed! 

It is an object of the present invention to provide a method and an apparatus for reliably and exactly sensing an 
object coming into a view field of an imaging device for a short time. 

It is a further object of the present invention to provide a method and an apparatus lor sensing an object which 
5 method and apparatus bring about no erroneous sensing resulting from the adverse effect of disturbance factors such 
as change of a weather or lighting conditions, reflected light or shadow. 

It is a still further object of the present invention to provide a method and an apparatus for sensing an object which 
method and apparatus serve to positively distinguish any portion where a local temperature is changed within the view 
field of the imaging device from the other portions. 
10 It is another object of the present invention to provide a method and an apparatus for sensing an object with highly 
reliance by providing a method for automatically creating a suitable reference background picture to deriving a differ- 
ence value for a short time. 

In the method and apparatus for sensing an object coming into the imaging field of an imaging device according to 
the present invention, the process is executed for storing in a storage unit data of the reference background picture 

is located within the imaging field, reading the data of the picture taken by the imaging device, comparing the data of the 
reference background picture with the read picture data at each pixel, outputting a difference value of each pixel, com- 
paring the difference value with a predetermined threshold value at each pixel, creating digital data of an object image 
based on the compared result, sensing if the object exists based on the binary data, and properly updating the data of 
the reference background picture stored in the storage unit. 

20 The other objects and advantages of the present invention will be apparent from the description that will follow with 
reference to the appended drawings. 
In the drawings 

Fig. 1 is a flowchart for describing an operation of an embodiment of the present invention; 
25 Fig. 2 is a flowchart for describing an operation of a further embodiment of the present invention; 

Fig. 3 is a flowchart for describing an operation of a still further embodiment of the present invention; 

Fig. 4 is a flowchart for describing ah operation of another embodiment of the present invention; 

Fig. 5 is a flowchart for describing an operation of another embodiment of the present invention; 

Fig, 6 is a flowchart for describing an operation of another embodiment of the present invention; 
30 Fig. 7 is a flowchart for describing an operation of another embodiment of the present invention; 

Fig. 8 is a flowchart for describing an operation of sensing an object by using a differential calculus; 

Figs. 9A, 9B, 9C t 9D, and 9E are pictures for describing a problem caused in sensing an object by using the differ- 
ential calculus; 

Fig. 1 0 is a view for describing the effect of a similarity used in sensing an object according to the present invention; 
35 Fig. 11 is ah explanatory view for sensing an object based on a difference in time according to the prior art; ^ 
Fig. 12 is a block diagram showing a system for monitoring a picture to which an embodiment of the present inven- 
tion is applied; 

Fig. 13 is a flowchart showing an embodiment in which the present invention is applied to sensing an object coming 
into a railroad crossing; 

40 Fig. 14 is a flowchart showing a process for creating a background picture according to an embodiment of the 
present invention; 

Fig. 15 is an explanatory view showing an example of an operation of sensing an object according to an embodi- 
ment of the present invention; 

Fig. 16 is an explanatory view showing an example of an operation of sensing an object according to an embodi- 
es mentof the present invention; 

Fig. 1 7 is a flowchart showing an embodiment in which the present invention is applied to a system for monitoring 
a pyrpclastic f low; 

Fig. 18 is an explanatory view showing an example of a sensing operation executed by the embodiment in which 
! the present inventions applied to a system for monitoring a pyrociastic~fI6w; ~~ 
so Fig. 1 9 is a flowchart for describing an operation of creating and updating a reference background picture according 
to a first embodiment of the present invention; 

Figs. 20A and 20B are diagrams for describing an effect caused by calculation of a median of pixel luminance val- 
ues; 

Fig. 21 is a flowchart for describing an operation of creating and updating a reference background picture according 
55 to a second embodiment of the present invention; 

Fig. 22 is a flowchart for describing an operation of a method for sensing an object according to an embodiment of 
the present invention; 

Fig. 23 is a flowchart showing an operation of an embodiment in which the present invention is applied to sensing 
an object coming into a railroad crossing; 
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Figs. 24A, 24B, 24C and 24D are explanatory views showing an operation of sensing an object through the effect 
of the differential calculus according to an embodiment of the present invention; 

Fig. 25 is a flowchart for describing an operation of sensing an object according to another embodiment of the 
present invention; 

s Fig. 26 is a flowchart for describing an operation of sensing an object according to yet another embodiment of the 

present invention; and 

Fig. 27 is an explanatory view showing an operation of sensing an object according to the embodiment of the 
present invention. 

10 Later, the description will be oriented to an embodiment of the present invention with reference to the drawings. 

Figs. 1 to 7 are flowcharts showing the operations of the embodiments arranged to sense an object intruding into 
the inside of a field of an imaging device. The methods according to these embodiments can be executed by the imag- 
ing device and a computer and under the control of a program configured to code a shown flowchart stored in a storage 
medium with a help of the computer. 

15 Before describing the methods shown in Figs. 1 to 7 t the description will be oriented to a system for monitoring a 
picture according to an embodiment of the present invention with reference to Fig. 12. 

Fig. 12 shows the picture monitoring system according to the embodiment of the invention. As shown, a TV camera 
121 captures a picture of an area to be monitored (monitoring field) and produces a video signal. The video signal is 
sent to a picture memory 124 through an input interface (l/F) 123 and a data bus 122. 

20 On the other hand, a CPU 1 26 operates to analyze the picture by using a working memory 1 25 in accordance with 
a program stored in a program memory 127 and then starts to sense an object. Based on the process of sensing an 
object, the program is executed to turn on an alarm lamp 1 30 through an output interface l/F 1 28 or display the picture 
on a monitor 131 through a picture output interface l/F 129. The program for executing the operations of the flowcharts 
of Figs. 1 to 7 is saved in the program memory 127. 

25 Along the flow shown in Fig. 1 , at a step 1 1 of inputting a picture, a picture is read from a TV camera. Then, at a 
step 12 of processing a difference, the operation is executed to calculate a difference of each pixel between the read 
picture and a pre-created reference background picture saved in a memory. Next, at a step 13 of saving code data, a 
. polarity of a difference value of a luminance is saved in a memory. At a next determining step 14, the operation is 
branched on the polarity. If the polarity is positive, the first binarization process (step 15) is executed. If it is negative, 

30 the second binarization process (step 1 6) is executed. The binarization process is executed to compare the difference 
value with a threshold value- 

These first and second binarization processes (steps 1 5 and 1 6) have the corresponding variable threshold values. 
Hence, if the quality of the change of a luminance against the reference background picture is variable depending on if 
the polarity is positive or negative, the exact binarization processes can be executed. In accordance with the binarized 

35 result, at a step 1 7, the object is allowed to be positively sensed. 

For example, assume that the change of the input picture against the reference background picture is brought 
about by the change of the weather from cloudy to fine or applying a ray of light to the object This change causes the 
polarity of the difference to be positive. Or, if the change is brought about by the shadow cast by the object so that the 
picture itself is made gloomier, this change causes the polarity of the difference to be negative. The variation depends 

40 on the polarity of the difference, that is, if it is positive or negative. Hence, since the prior art operates to simply binarize 
the absolute value of the difference with a unique threshold value, it does not cope with the wide variation. 

However, the method shown in Rg. 1 is arranged to change the direction of the change, that is, the threshold value 
depending on if the polarity of the difference is positive or negative. Hence, this method enables to flexibly cope with the 
variation, so that the object may be exactly sensed. 

45 In the flow shown in Fig, 2, a process at a step 24 is added to the process shown in Rg. 1 . The process at the step 
24 is executed to calculate a similarity of a local luminance distribution pattern between the input picture and the refer- 
ence background picture. The flow of Rg. 2 is arranged to contain the similarity as the binarization conditions in the first 
and the second binarization processes (steps 26 and 27). Hence, this processing flow can more flexibly cope with the 
variation of the input picture against the reference background picture, so that more exact sensing is made possible. 

so The description about the operations of steps 21 to 23, 25 and 28 is left off, because they correspond to the steps 11 
to 13, 14 and 17 of Fig. 1 , respectively. 

Herein, the similarity represents a correlation of the input picture to the reference background picture about the pat- 
tern of a local luminance distribution. In Fig. 10, assume that the whole input picture is made gloomier as compared to 
the reference background picture 101 as indicated by a numeral 102 by the change of illuminance in the imaging envi- 

55 ronment. in thte case, patterns of the background picture 1 01 such as white lines and railroad are left in the input picture 
1 02 as they are. Hence, a high similarity takes place between a local area of the background picture 1 01 , for example, 
an area 1 04 and a local area 1 05 of the input picture 1 02. 

On the other hand, unlike the reference background picture, as indicated by 103, on the area of the input picture 
where an object (vehicle) 106 is located, the patterns such as white lines and railroad are disappeared, because the 
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object 106 hides the background. Hence, a low similarity takes place between the areas 104 and 106. 

In the flow of Fig. 3, at the next step 35 to the binarization step 34, the similarity is calculated. At a step 36, the cal- 
culated similarity is compared with the reference value. Depending on whether or not the similarity is higher than the 
reference value, two areas A and B having the different qualities from each other, that is, the area where the luminance 
5 of the background picture is changed and the area where the object is located, are separated from each other. 

The calculation of the similarity at step 35 is executed only for the pixels at which the difference value exceeds the 
predetermined threshold value in the binarization process at the step 34. This makes it possible to reduce an amount 
of calculation to a minimum. 

In the flow of Fig. 4, for sensing an object, the process of estimating the similarity is added to the process shown in 
io Fig. 2. If the polarity of the difference is positive, the binarization process is executed at a step 45. Then, at a step 46, 
the similarity of the form of a local luminance distribution is calculated between the input picture and the reference back- 
ground picture. At a step 47, the similarity is compared with the reference value. As shown in Fig. 9D, if there exists such 
an area 94 as being brighter by light applied from a lamp, it is determined that the polarity of the difference is positive 
and the similarity is high. Hence, at a step 48, it is not determined that any object exists in the area 94. Conversely, if 
is the polarity of the difference is positive and the similarity is hot so high (103 of Fig. 10), at a step 49, it is determined 
that there exists an object in the area 94. 

That is, as indicated by the area 94 of Fig. 9D, in the area that is made brighter by light, though the luminance is 
made higher, the patterns on the road are left. Hence, like the local areas 1 04 and 1 05 of Fig. 1 0, the similarity is made 
higher. 

20 The use of this similarity, therefore, makes it possible to reliably distinguish the object from the area 94 that is made 

brighter by a ray of light applied from the lamp. 

Next, in the flow of Fig. 5, like Fig. 4, for sensing an object, the process of estimating the similarity is added to the 

process shown in Fig. 2. If the polarity of the difference is negative, at a step 55, the binarization process is executed. 

At a step 56, the process is executed to calculate a similarity of the pattern of a local luminance distribution between 
25 the input picture and the reference background picture. At a step 57, the similarity is compared with the reference value.- 

As shown in Fig. 9E, if there exists an area 96 that is made dark by the shadow of the object/it is determined that the 

polarity of the difference is negative and the similarity is high. Hence, at a step 58, it is not determined that any object ' 

exists in the, area 96 (shadow). Conversely, if the polarity of the difference is negative and the similarity is nbt so high,- 

at a step 59, it is determined that there exists an object in the area 96. 
30 That is, in the area that is made dark by the shadow of the object itself, as indicated by the area 96 of Fig. 9E f 

though the luminance is low, the patterns on the road are left. Like the local areas 104 and 105 in Fig. 10, hence, the 

similarity is made higher. 

The use of this similarity, therefore, makes it possible to reliably distinguish the object from the area 96 that is made 
dark by the shadow. 

35 In the flow of Fig. 6, there is introduced a process (step 67) of estimating a similarity of the pattern of a local lumi- 
nance distribution between the input picture and the reference background picture. This added process makes it possi- * 
ble to sense an area where a temperature locally goes up. ~ 
At a step 61 , an object is imaged by an infrared TV camera. The captured video signal is saved as the input picture 
in a memory. At a step 62, at each pixel, the read input picture is compared with the reference background picture 

40 imaged by the same infrared TV camera and pre-stored in the memory for sensing the difference value between both 
of the pictures and its polarity. Then, at a step 63, the polarity data is stored in a memory. If, at a step 64, the polarity is 
determined to be positive, at a step 65, the difference value calculated at the step 62 is compared with a predetermined 
reference value (threshold value) so that the binarization process may be executed. At a step 66, the operation is exe- 
cuted to calculate the similarity between the reference background picture and the input picture. If, at a step 67, the sim- 

45 ilarity is determined to be low, at a step 68, it is determined that there exists in the input picture an area (object) where 
a temperature goes up. 

That is, a difference of a luminance distribution takes place between the area where the local temperature goes up 
and the area where it keeps stable. The similarity of the form of the luminance distribution between the input picture and 
the reference background picture is made low. For example, assume that the method for sensing an object as shown in 
so Fig. 6 is applied to sensing a pyroclastic flow. If a lava dome is formed by the pyroclastic flow, the temperature abruptly 
goes up in the local area of the lava dome: Hence, the similarity is made lower. On the other, hand, for example, as to 
the clouds that are made warmer by the pyroclastic flow, the temperature goes up uniformly for the whole clouds. 
Hence, the similarity is made higher. 

Hence, the process (step 67) of estimating the similarity is added to the process of Fig. 1 in order that the warmed 
55 clouds may be distinguished from the pyroclastic flow. 

In the flow of Fig. 7, there is introduced a process (step 77) of estimating a similarity of the form of a local luminance 
between the input picture and the reference background picture. This process makes it possible to sense a local area 
where the temperature goes down. 

At a step 71. an object is imaged by an infrared TV camera. The video signal captured by the TV camera is read 
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as an input picture in a memory. At a step 72, the reference background picture taken by the infrared TV camera and 
pre-stored in a memory is compared with the read input picture at each pixel so that the difference value between both 
of the pictures and its polarity may be sensed. At a step 73, the polarity data is stored in a memory. If the polarity is 
determined to be negative at a step 74, at a next step 75, the difference value calculated at the step 72 is compared 

s with the predetermined reference value (threshold value) so that the binarization process may be executed. At a step 
76, the process is executed to calculate a similarity between the reference background picture and the input picture. If, 
at a step 77, the similarity is determined to be low, at a step 78, it is determined that there exists an area (object) whose 
temperature goes down in the input picture. 

That is, a difference of a luminance distribution takes place between the local area whose temperature goes down 

10 and the local area whose temperature is inhibited to go down. The similarity of the pattern of a luminance distribution 
is made lower between the input picture and the reference background picture. For example, consider that the method 
for sensing an object as shown in Fig. 7 is applied to sensing a pyrociastic flow. If the lava dome is fell down, the tem- 
perature of the lava dome is abruptly made lower, because the heat source of the lava dome is dropped. It means that 
the similarity is lower. On the other hand, for example, as to the clouds warmed by the pyrociastic flow, if the application 

is of heat to the clouds by the heat source is reduced, the whole.cfouds are substantially uniformly cooled down. Similarly, 
the temperature of the clouds may uniformly go down. Hence, the similarity is made higher. 

At a step 77 of estimating the similarity, therefore, the process makes it possible to separate only the object (area) 
whose temperature locally goes down from the other objects in sensing the objects. 

Later, the method for sensing an object according to the present invention will be described in detail along the illus- 

20 trative embodiments. 

The below-described embodiments are arranged so that the present invention is applied to a picture monitoring 
system for sensing an object such as a walker or a car coming into a railroad crossing. A TV camera is installed at a 
site where the camera can take a look at the whole railroad crossing. The TV camera produces a video signal repre- 
senting the picture taken by the camera itself. The video signal is processed for sensing an object. The description 
25 about the embodiment will be expanded along the flowcharts. The operation of the flowchart about the embodiment is 
executed by the system shown in Fig. 12. 

Fig. 13 shows an object sensing process executed by the CPU 126 according to an embodiment of the invention. 
The process executed by this embodiment is a combination of the processes shown in Figs. 4 and 5 to which a process 
of updating a background picture is added. 
30 When the process of Fig. 13 is started, at a step 1 41 of inputting a picture, a video signal picked by a camera 121 
is read as the image data f(x* y) consisting of 320 x 240 pixels, where x andy are coordinates of pixels on the screen 
^) and f(x, y) represents a luminance value of a pixel at the coordinates (x, y). 

Next, at a determining step*142, it is determined whether or not the new reference background picture is created. 
■ 5 ln order t0 create and update this reference background picture, the CPU 126 executes the process shown in Fig. 

^Pj 35 14 in parallel to the flow of Fig. 13. 

Entering the flow of Fig. 1 4, at first at a step 161 , the input picture are captured from the TV camera at regular inter- 
vals of 0.2 second. When the frames of the captured picture reach 20, those 20 frames are saved (stored) in a memory. 
Next, at a step 162, at each pixel, the median of each of the 20 frames is calculated by the following expression (1) 
_ for creating the background picture data r(x, y). 

40 


r (x, y) = med {£ (i) U, y) } i i \ 


45 


wherein f®(x, y) represents the picture data for 20 frames and i = 1 , 2 20. 

In the expression (1), med { } represents a median. The pixels at the same location of the 20 frames are ranged 
in greater order of their luminance values. Taking the pixel of the tenth (middle) value, the pixel is assumed as the back- 
ground picture data r(x, y). At a step 163, this expression (1) is calculated for deriving the new reference background 
so picture. 

Hence, the reference background picture 1 63 is made equivalent to the picture data r(x, Y). 

In this embodiment, only 20 frames of the picture are sampled at regular intervals of 0.2 seconds. The frame period 
of the imaging device such as the normal TV camera is 1/60 second. To read the picture on which the median calcula- 
tion is done, of 60 frames per one second output from the TV camera, any number of frames may be read at proper 
55 intervals if necessary. That is, the output frames of the imaging device are properly thinned out for reading them. Or, the . 
picture may be read at a longer period than the output period of the frame of the imaging device. We will further explain 
this matter later. 

Going back to the process of Fig. 1 3, if the creation of the background picture is terminated, at a step 1 43, the exist- 
ing reference background picture stored in the memory is updated to a new reference background picture that has been 
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just created. Then, the operation goes to the next step 144. If the creation of the new reference background picture is 
not terminated, the operation goes to the step 1 44 of processing a difference. At a step 1 44, the input picture is divided 
into blocks each consisting of 2 x 2 pixels. The overall picture is divided into a block area f n (i, j) consisting of 160 x 120 
pixels. Then, an average luminance Af of each block area is derived, where n is a block number and i, j represents a 
coordinate location of a pixel inside of the block. 

Further, for the reference background picture data r(x, y), likewise, the overall picture is divided into a block area 
r n (i, j) consisting of 160 x 120 pixels. Then, an average luminance Ar of each block area is derived. The average lumi- 
nances Af and A, of each block area are calculated by the following expression (2); 


IJeB 


15 A r=\k X r nV'J) 

iJeB 

where B represents the subject block area, [B] represents a pixel number contained in the area B. (This embodiment 
20 uses a block consisting of 2 x 2 pixels. Hence, i, je B represents 0 ^ i ^ 1 and 0 ^ j ^ 1 and [B] is four.) 

Next, a difference value e n of an average luminance between the input picture and the reference background pic- 
ture is derived at each block by the expression (3). 


e n = |Af-ArJ (3) 
(n = 1,2,3...) 


At the succeeding step 145 of saving the polarity data, the polarity data S n of the difference is derived by the 
30 expression (4) and then is saved in a memory. 

S " { -l (Af £ Ar) * ( 4 ) 

35 


Next, at a determining step 146, the operation proceeds to the first or the second binarization step 147 or 152 
according to the polarity data S n . 
40 These binarization processes are executed by comparing the difference value ^ with a predetermined threshold 
value. The threshold value for the positive polarity S m that is, the threshold value th+ at the step 147 is derived by the 
expression (5). 


45 


50 


th + =H + + K + a + (5) 

.where 


+v2 


X < e <rn + ) 


where ^ represents a difference value between the input pixel and the background picture at the d-th block (see the 
55 expression (2)), 

n + is an average difference value of the blocks having positive difference values of the blocks predetermined to con- 
tain no object this value representing an average level of noise contained in the picture, 

a + is a standard deviation of the difference values of the blocks having positive difference values of the blocks pre- 
determined to contain no object, this value representing how variable the level of the noise contained in the picture is, 
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and 

K+ is a adjustable parameter and [R+] is a number of pixels contained in an area wherein the polarity Sn is positive 
and no object exists. 

Further, at the step 152, likewise, the threshold value th* for the negative polarity Sn is derived by the following 
expression (6). 

th" = n" + K" a" (6) 

, where 


•\2 


[*'] 1 IR'] 


where K' is an adjustable parameter and [R] is a number of pixels contained in an area having a negative polarity and 
no object. 

At the steps 1 48 and 1 53 of calculating the similarity, for each block, the similarity y n is calculated by the following 
20 expression (7). 

_ deD n ijeB - 

25 > ' ■ Yn ~ [ej-iDj . 

where D n represents an area near an n-th block and [ ] represents a number of elements. 

At steps 1 49 and 1 54 of determining the similarity, the adjustable parameters T* and T are set as the threshold val- 
30 ues for these steps. On these parameters, it is determined whether or not the similarity y n is higher than the threshold 
value. 

Concretely, at the determining step 1 49, the similarity y h is compared with the threshold value T. If the similarity y n 
exceeds the threshold value T 1 ", at the step 150, it is determined that the area that is lit by the lamp is sensed as indi- 
cated in Fig. 9D as the area 94 from which the light is reflected. Only if the similarity y n is smaller than or equal to the 
35 threshold value T+ at the step 1 49, it is determined that the.object is sensed at the step 151. 

Likewise, at the determining step 154, the similarity y n is compared with the threshold value T. If y n exceeds the 
threshold value T, at the step 1 55, it is determined that the area that is made dark by the shadow of the object is sensed 
as indicated in Fig. 9E as the shadow area 96. Only if y n is smaller than or equal to the threshold value T, at the step 
1 56, it is determined that the object is sensed. 
40 Then, when the object is sensed at the step 151 or 156, the CPU 126 operates to turn on the alarm lamp 130 
through the output l/F 128 or display the picture on the monitor 131 through the picture output l/F 129 for reporting to 
the user that the object is sensed. 

As mentioned above, the method according to this embodiment is capable of positively sensing only an object 
within the field of the imaging device. The concrete arrangement to which the method for sensing an object as shown 
45 in Fig. 1 3 will be described with reference to Figs. 15 and 16. 

In Fig. 15, on an input picture 171, at dusk, a car 171 A is stopped near a railroad crossing. The shadow 171B of 
the car 171 A extends to the inside of the crossing. If this input picture 171 is determined by the conventional method, 
as indicated on a picture 1 73, it is erroneously determined that there exists an object 1 73A coming into the crossing. 
On the other hand, according to the embodiment shown in Fig. 1 3, at first, the process shown in Fig. 1 4 is executed 
so to obtain a reference background picture 172 and ipdate it to a new background picture every about four seconds. 

Next, since the portion of the shadow 1 71 B makes the code data S n negative, the operation goes to the determin- 
ing step 154 at which the similarity between the reference background picture 172 and the input picture 171 is deter- 
mined. 

In the area where the car 171 A exists, since the car 171 A hides a portion of the background, the similarity is not 
55 made so high. At the step 1 56 shown in Fig. 1 3, therefore, this area 1 71 A is determined to actually have an object On 
the contrary, in the area of the shadow 171B, nothing hides the background. Hence, the similarity is made high. As a 
result, at the step 1 55 of Fig. 1 3, it is determined that it is not an object but the shadow. 

According to this embodiment, therefore, as shown by the picture 174, the car 174A is clearly distinguished from 
the shadow 174B, so that only an object coming into the crossing is positively sensed. In this case, though the shadow 
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1 74B appears, it is determined that no object comes into the crossing. 

Next, Fig. 16 shows an input picture 181 where a ray of light 1 81 B applied by a head lamp of a motorcycle 181 A is 
intruded into the railroad crossing before the motorcycle enters into the crossing. In this case, if the input picture 181 is 
determined by the conventional method, as shown by the picture 183, it is erroneously determined that there exists an 
object 183 A coming into the crossing. 

On the other hand, according to the embodiment shown in Fig. 1 3, at first, the process shown in Fig. 1 4 is executed 
to obtain a reference background picture 1 72 and update it to a new picture every about four seconds. 

Next, since the polarity data S n is positive in the ray of light 1 81 B applied by the head lamp, the operation goes to 
the step 149 of determining the similarity. At this step 149, it is determined whether or not the similarity between the 
latest reference background picture 1 72 and the input picture 1 81 is high. 

In the area where the motorcycle 181 A exists, since the motorcycle 181A hides a portion of the background, the 
similarity is not made so high. Hence, at the step 151 , it is determined that there exists an object. On the other hand, in 
the area of the ray 181 B, since it does not hide any portion of the background, the similarity is made high. As a result, 
at the step 1 50, it is determined that it is not an object but a mere bright portion. 

According to this embodiment, therefore, as shown by the picture 184, the motorcycle 174A and the ray applied 
from its head lamp are clearly distinguished from each other as the objects 1 51 and 1 50 shown in Fig. 1 3. It means that 
only the object coming into the railroad crossing can be positively sensed. In this case, though the portion of the ray 
184B appears, it is determined that there exists no object coming into the railroad crossing. 

Next, the description will be oriented to a method according to another embodiment of the present invention. 

The arrangement of this embodiment is application of the present invention to monitoring a pyroclastic flow in an 
active volcano. In the apparatus for monitoring a picture as shown in Fig. 12, an infrared TV camera (infrared imaging 
device) is used for the camera 121. The infrared TV camera yields the input picture on which a luminance is made larger 
in a warmer portion of an object. In the system shown in Fig. 12, the CPU 126 executes the process shown in Fig. 17. 

In Fig. 17, as depicted in brackets, the routines from steps 191 to 199 and from steps 201 to 203 are the same as 
those from the steps 141 to 149 and from the steps 152 to 154 shown in Fig. 13. Hence, these routines will be briefly 
described. The processing operation goes to any one of the step 199 of determining the similarity and the step 203 of 
the same function, depending on the luminance, that is, the polarity of the difference of the temperature between the 
input picture and the reference background picture. At the steps 1 99 and 203, the adjustable parameters T + and T are 
used as the threshold values for determining the similarity. If an area has a positive polarity and a similarity that is 
smaller than or equal to the threshold value, at the step 200, it is determined that the area (object) has a higher temper- 
ature. If the area has a negative polarity and a similarity that is smaller than or equal to the threshold value, at the step 
204, It is determined that the area (object) has a lower temperature. 

The operation of the method according to this embodiment will be described with reference to Fig. 18. An input pic- 
ture 21 1 is a picture captured at an instance when a pyroclastic flow takes place. A numeral 212 denotes a reference 
background picture. 

Then, after the process of Fig. 17 is executed for the picture shown in Fig. 18, the resulting picture is made to be 
the picture 213. On this picture, a portion whose temperature is made higher by the pyroclastic flow js allowed to be 
positively distinguished from the portion whose temperature is made lower after the pyroclastic flow is flown out. 

That is, since the difference of the luminance distribution pattern takes place between the area whose local tem- 
perature is higher and the other areas whose temperatures keep stable, the similarity of the form of the luminance dis- 
tribution is made lower. 

In the portion where a lava dome is formed by the pyroclastic flow, the local temperature abruptly goes up. Hence, 
the similarity is made lower. On the other hand, for example, in the clouds that are made warmer by the pyroclastic flow, 
the whole temperature uniformly goes up- Hence, the similarity is made higher. 

Hence, the addition of the step 199 of estimating the similarity makes it possible to clearly distinguish the area 
(pyroclastic flow) whose temperature goes up from the other area at the step 200 if there exist portions such as the 
heated clouds. 

On the other hand, in the area whose local temperature goes down, the difference of the luminance distribution pat- 
tern takes place between that area and the area whose temperature keeps stable. Hence, the similarity of the pattern 
of the luminance distribution is made lower. 

If the lava dome is fell down, it means that the heat source is dropped. The temperature of the relevant area is made 
abruptly lower. Hence, the area is made to have a lower similarity. On the other hand, in the area such as the clouds 
heated by the pyroclastic flow, if the heat applied by the heat source is reduced, the overall area is substantially uni- 
formly cooled down, that is, lowers its temperature. Hence, the similarity is made higher. 

The addition of the step 203 of estimating the similarity, therefore, makes it possible to clearly distinguish the area 
whose temperature is made lower (pyroclastic flow is flown out) from the other area at the step 204 if there exist portions 
such as the clouds whose temperature is made lower. It means that the cooled area can be distinctly sensed. 

In addition, it goes without saying that the present invention is not limited to the above-mentioned application but 
may be applicable to another system such as a system for monitoring an intruder to a prohibitive zone such as plant 


9 


EP 0 749 098 A2 


facilities. 

According to the embodiment as described above, the sign of the difference value is used for overcoming the con- 
ventional shortcoming of the erroneous sensing resulting from the change of the background caused by the variation of 
illuminance. This thus makes it possible to lower the adverse effect caused by the change of the environment such as 
s change of illuminance of the monitoring field, thereby being able to accurately sense an object. Further, the similarity of 
the pattern of the luminance distribution is used for determining how much of degree the patterns are saved. Hence, if 
a phantom area caused by the change of the illuminance takes place though no actual object exists, such an area is 
allowed to be positively distinguished from an actual object when sensing an object 

As noted above, the method according to the present invention may be reliably applied to such an area as requiring 
10 high reliance like a railroad crossing. It means that this method may make great contribution to widening the applied 
field of the apparatus for monitoring a picture. 

In the embodiment as described above, for generating the reference background picture, the method shown in Fig. 
14 as wejl as another method may be used. The reference background picture may be produced for calculating a 
median of plural picture frames as well as calculating an average value of plural frames though the latter method needs 
is more frames. Moreover, another statistical method may be used therefor. A method for generating a reference back- 
ground picture by calculating a median will be described below with reference to the drawings. 

. The method according to the embodiment shown in Fig. 19 includes the step 301 of receiving a picture, a step 302 
of storing the picture, a step 303 of determining whether or not a predetermined number of frames have been saved, a 
step 304 of calculating a median for each pixel of the picture saved when the storage for the predetermined number of 
20 frames is terminated, and a step 305 of updating the picture calculated at the step 304 to a new reference background 
picture. 

In Fig. 19, at the step 301, the picture is read from the TV camera at a rate of 30 frames per second or a higher / 
sampling rate. Then, the read picture is stored in the memory at the step 302. 

Afterwards, if, at the step 303, the number of the stored frames reaches a predetermined number N (N is a positive 
25 integer), at thp step 304, a median value of a luminance for each pixel is calculated on the picture signal for the stored 
N frames. Then, at the step 305. the picture composed to have the median value as the luminance of each pixel is 
updated to a new reference background picture and then saved in the memory. 

As this result, the method of this embodiment is capable of positively securing a quite exact reference background 
picture without having to set a large value as the frame number N (for example, N = 300). Trie ground will be described 
30 below. 

Remarking a pixel (x, y) of the picture whose N frames are stored, the N frames are ranged in numerical order. The * . 
change of the luminance of the pixel is made to be as shown by 310 in Fig. 20A. Herein, (x, y) represents a pixel location 
(coordinates) within a frame. It means that the picture data composed of 320 x 240 pixels has the values of x = 1 to 320 
andy = 1 to 240. 

35 Next, about the pixel (x, y) at one location, a median value of the luminance is calculated. The sentence "A median 
value of the luminance is calculated" means that pieces of data ranged in an order are rearranged in order of magnitude 
from a smaller to a larger (ascending order) or from a larger to a smaller (descending order), so as to select a central 
value at the middle of the order. 

As a result of calculating a median of the luminance of each pixel as shown in Fig. 20A, the distribution as shown 

40 by a characteristic 320 in Fig. 20B takes place. The median value of the pixel (x, y) is the N/2-th luminance value M, that 
is, a median value M. 

If no change of the picture takes place over N frames, the luminance value of the same pixel of each frame is made 
equal to the median value M. In this case, the characteristic 320 is made to be a horizontal line. 

Then, assume that an object is caught or any noise is mingled on L frames (1 < L < N) of the stored N frames 
45 though the object or noise is not on the other frames rather than the L frames. The pixels composing the object or the 
pixels mingled with the noise is made to have a different luminance value from the pixels of the picture that does not 
suffer from any change, that is, a different median value M therefrom. By rearranging the luminance values of the pixels 
at the same location of the N frames in descending order, if the frame number L is equal to or smaller than N/2, the pixel 
is located at both ends of the characteristic 320 of Fig. 20 B spaced from the center location (N/2). 
so When the luminance values of the pixels composing the object or mingled with noise are smaller than those of 
these pixels that do not suffer from any change, the pixels are located in the left hand (toward a value of 1) from the 
center location (N/2). Conversely, when the former is larger than the latter, the pixels are located in the right hand 
(toward a value of N) from the center location (N/2). 

As the number L of the frames where an object or any noise is caught is made greater, the pixels appearing close 
55 to the center location (N/2) is made more numerous. While the frame number L is N/2 or less, no substantial influence 
effect is applied to the median value M. 

That is, the influence applied by the object or any noise takes place in the area around the center location (N/2) 
close to the median value M only when the pixels where the object or any noise is caught stay at the same location over 
at least a half of a number N of stored frames. 
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In principle, however, a moving object or noise is quickly moved or disappears. H means that the number of frames 
where the object or noise is caught is not so great. 

Hence, by calculating a median value, the influence of the moving object or noise may be restricted without having 
to read so many frames N. This calculation of the median value makes it possible to obtain a quite exact reference back- 
5 ground picture. In addition, the calculation of the median value just needs a half or less of the stored frames if an object 
is caught in some of the frames. Hence, by lowering a sampling rate, the frames to be stored may be reduced in number. 

Next, the method according to another embodiment of the invention shown in Fig. 21 is arranged to add an object 
sensing step 402 and a determining step 403 after the step 401 of receiving a picture. After rt is determined whether or 
not an object exists at the steps 402 and 403, the same process as described about the method of the embodiment 
10 shown in Fig. 1 9 is executed only for the picture that is determined to have no object In addition, the description about 
the processes at the steps 401 and 404 to 407 is left out, because those processes are the same as those at the steps 
301 to 305., At the step 402, by using a differential calculus, it is determined whether or not an object comes into an 
imaging field. Then, the picture that is determined to have no object at the step 403 is stored in the memory at the step 
404, Next, if the number of stored frames reaches the predetermined number, the processes at the step 406 of calcu- 
15 lating a median value and the step 407 are executed to calculate a median value (step 406) and update a reference 
background picture (step 407). 

According to this embodiment, therefore, it is guaranteed at the step 403 that a new background picture used for 
updating the reference background picture at the step 407 does not have any object coming into the picture. This thus 
makes it possible to greatly reduce the number N of frames required for calculating a median value to a small value such 
20 as N = 6 as compared to the value used in the embodiment shown in Fig. 19. 

Next, in the method according to another embodiment shown in Fig. 22, the processes done by the embodiments 
,shown in Figs. 1 9 and 21 are allowed to be executed in parallel. At the steps 305 and 407, that is, two-system processes 
of updating a reference background picture, the method of this embodiment is capable of constantly yielding the latest 
reference background picture. 

25 In the embodiment shown in Fig. 19, the number of frames to be stored is about 300, which is far more numerous 
than that of the embodiment shown in Fig. 21. Hence, the interval for updating the reference background picture is 
made considerably wide, ft means that the freshness of the reference background picture is not necessarily satisfactory. 
' On the other hand, in the embodiment shown in Fig: 21 , if yes at the step 403 of determining whether or not an 
object exists, the process at the step 407 of updating a reference background picture is not executed. In this case, it is 
30 possible to make no use of such a special advantage as reducing the necessary number of frames. 

The method according to the embodiment shown in Fig. 22, however, can make use of both advantages of the 
embodiments shown in Figs. 1 9 and 21 and allow the processes for these embodiments to use the corresponding latest 
reference background picture. Hence, the reference background picture can be made more exact. 

Fig. 23 shows a series of processes from a process of creating a reference background picture to a process of 
35 sensing an object, which processes are executed by the CPU 126 in the system shown in Fig. 12. This embodiment is 
based on the method shown in Fig. 19. 

At a step 501 of reading a picture, a video signal is read as picture data consisting of 320 x 240 pixels from a TV 
camera 121 . Then, at a step 502 of storing a picture, the picture data is stored in a picture memory 124. 

At a determining step 503, if the number of the frames being stored reaches a predetermined value N 1 (for exam- 
40 . pie, N 1 = 300 = N), the operation is branched into a step 504 of calculating a median. If it does not reach the value N1, 
the operation is branched into a step 506 of sensing an object. 

At the step 504 of calculating a median, the process is executed to rearrange the stored N frames as shown in Fig. 
20 for each pixel and calculate a median value M by the following expression (8). 

45 ' < 

r(x,y)=med{f U) {x,y)} . "(8i- 

where f®(x, y) represents the picture data composed of 300 frames and i takes a value of i = 1 , 2 300. in addition, 

so the expression (8) is the same as the case that the number N of the frames in the expression (1) is 300. 

At the step 505, the median value derived at the step 504 is stored in a storage area for the reference background 
picture allocated to the picture memory 124. 

Next, at the step 506, the process is executed to derive a difference between the picture data read at the step 501 
and the reference background picture stored in the storage area of the picture memory 124. If the picture data has a 
55 portion with a larger difference than a predetermined threshold value, the portion is sensed as an object. 

At first, the reference background picture as shown in Fig. 24A is compared with the picture containing an object to 
be sensed as shown in Fig. 24B. At each pixel located at the same coordinates of both of the pictures, the difference of 
the luminance is derived. The picture composed on the difference value is made to be as shown in Fig. 24C. Then, the 
difference value is given a predetermined threshold value and then is made binarized. As a result, the picture shown in 
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Fig. 24D Is extracted. 

On the other hand, if no object exists, it does not appear as the extracted picture. Hence, only the object is allowed 
to be sensed. 

At a step 507, it is determined whether or not an object exists based on the result given at the step 506. If any object 
s .exists, at a step 508, an alarm is issued. If no object exists, the operation is branched into the step 501 of reading a 
picture. 

In turn, Fig. 25 shows an applied one of the method described with reference to Fig. 21 . Each figure in brackets of 
Fig. 25 indicates the step of another Figure at which the same process as the figure-attached step is executed. 

In the embodiment shown in Fig. 25, the processes at the steps 601 to 605 the same as those at the steps 501 to 
10 505 in Fig. 23. Hence, the description about these steps is left off this specification. 

At the step §06, like the step 601 , the video signal is read as the picture data consisting of 320 x 240 pixels from 
the TV camera 121 to the picture memory 124. 

At a step 607 of sensing an object, if a new reference background picture is created at a step 612, the new one is 
used. If not, a reference background picture created at the step 605 is used for sensing an object. 
is . At a step 608, if it is determined that no object exists at the step 607 of sensing an object, the operation is branched 
to a step 609, while if it is determined that any object exists, the operation is branched to a step 613. 

If it is determined that no object exists, at the step 609, the picture obtained at the step 606 is stored in the memory. 

At a determining step 610, if the number of f rames being stored reaches a predetermined number N 2 (for example, 
N 2 = 6), the operation is branched to a step 61 1 of calculating a median value. If the number does not reach the prede- 
20 termined value N 2 , the operation is branched to a step 606 of reading a picture. 

At the step 61 1 of calculating a median value, the operation is executed to rearrange the predetermined number N 2 
of the stored frames as shown in Fig. 20 for each pixel and calculate a median value by the above-indicated expression 
(8). - 

The median value derived at the step 61 1 is stored in a storage area for a reference background picture allocated 
25 to the picture memory 124 at a step 612. 

Fig. 26 shows an applied one of the method described with reference to Fig. 22. 

The processes at the steps 701 to 705 are the same as the steps 501 to 505 of Fig. 23. Hence, the description 
about these processes is (eft off the specif ication. 

In succession, at a step 706, the process of reading a picture is executed. This is the same as the process of the 
30 step 501 . And, at a step 707, the process of sensing an object is executed. This is the same as the process of the step 
506. Then, a first and a second systems of processes are executed in parallel. The first system of process is the same 
as the method shown in Fig. 19. The second system of process is the same as the method shown in Fig. 21 . 

The processes of steps 708 to 710 corresponding to the first system of process are the same as the processes of 
the steps 502 to 504 shown in Fig. 23. Next the processes of steps 712 to 715 and 71 7 corresponding to the second 
35 system of process are the same as the processes of the steps 608 to 611 shown in Fig. 25. Hence, the description 
about those steps is not left off the specification: In step 71 1 , a picture obtained by the first system of the process is 
taken as a candidate A of the reference background picture. In step 716, a picture obtained by the second system of 
the process is taken as a candidate B of the reference background picture. 

At steps 71 8, 71 9 and 720, the processes are executed to select an earlier one of a reference background picture 
40 A obtained at the step 71 1 for the first system of process and a reference background picture B obtained at the step 716 
for the second system of process and replace the previous reference background picture stored in the memory with the 
selected one. A number N 0 of frames read at the step 703, a number N-| of frames read at the step 709, and a number 
N 2 of frames read at the step 714 are set to have the relation of N 0 ^ N 1 ^ N 2 . If the relation of N 0 > N-j > N 2 is set, a 
proper number of frames to the imaging condition are sampled so that an object may be efficiently sensed. For example, 
45 a recommended value of N 0 is N 0 = 500. 

In turn, the description will be oriented to the sensing results given by those embodiments with reference to Fig. 27. 

In Fig. 27, a picture 801 is an example of picture data that is taken by the TV camera 121 (see Fig. 12) and read at 
the picture reading step 501 . 

For such a picture, according to the embodiment shown in Fig; 23, unless a truck in the picture stays at one site for 
so about five or more seconds (N = 300 at a frame frequency of 60 Hz), an exact reference background picture where no 
object (truck) appears can be positively obtained as shown by the picture 802 of Fig. 27. 

In the embodiments shown in Figs. 25 and 26, unless the object is erroneously sensed for at least three frames, as 
shown by the picture 802 of Fig. 27, an exact reference background picture where no object (truck) appears can be 
obtained. 

55 According to the above-mentioned embodiments, therefore, if the frames required for creating the reference back- 
ground picture are reduced in number, the exact reference background picture can be positively obtained. Hence, even 
in the conditions like an outdoor place where an environmental change such as variable illuminance, the latest refer- 
ence background picture with little delay may be used for sensing an object. This makes it possible to offer a highly reli- 
able monitoring system. 
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In addition to the above-mentioned applications, the present embodiment may be applicable to a system for moni- 
toring an intruder to plant facilities or prohibitive zones. 

According to the present embodiment, the calculation of a median is used for creating the reference background 
picture. Hence, the method of this embodiment can overcome the drawback the conventional technique of creating the 
5 reference background picture has involved and is endurable in light of noise. Further, the method of this embodiment 
needs a smaller number of frames for creating the reference background picture as compared with the method 
arranged to take an average of each pixel therefor. Hence, the method of this embodiment offers a highly reliable mon- 
itoring system with the latest reference background picture. 

In addition, an object sensing method according to the present invention may use luminance data as well as color 
10 data of the video signal. 

Claims 

I. An apparatus for sensing an object within an imaging field of an imaging device, comprising: 

15 ■ ■ 

storage means (124) for storing data of a reference background picture fitted in said imaging field; 
means (123) for reading data of a picture taken by said imaging device; 

means (1 25, 1 26) for comparing the data of said reference background picture data with said read picture data 
at each pixel and outputting a difference value of each pixel; 
20 means (1 26) for updating the data of said reference background picture saved in said storage means; 

means (1 26) for comparing said difference value with a predetermined threshold value at each pixel and pro- , 

ducing binarization data about said object based on the compared result; and 

means (126) for sensing whether or not said object exists based on said binarization data. 

25 2. An apparatus as claimed in claim 1 , wherein said means for outputting said difference value contains means for 
sensing a polarity of said difference value and said means for producing digital data contains first binarization 
means having a threshold value for a positive polarity of said difference value and second binarization means hav-'-v 
ing a threshold value for a negative polarity of said difference value. 

30 3, An apparatus as claimed in claim 2, further including means for comparing the data of said reference background " 
picture with said taken picture data at each block consisting of a predetermined number of pixels and calculating a 
similarity of said data between the blocks of said both of data, and said means for sensing whether or not said 
object exists contains means for sensing intrusion of said object based on said similarity and said binarization data. 

35 4. An apparatus as claimed in claim 3, wherein the data of said reference background picture and said current taken; 
picture data contain data about luminance of plural pixels composing 

5. An apparatus as claimed in claim 4, wherein said imaging device includes a television camera (121). 

40 6. An apparatus as claimed in claim 4, wherein said imaging device includes an infrared imaging device. 

7. An apparatus as claimed in claim 1 , wherein said means for reading said data operates to read said picture data 
having a predetermined number of frames from said imaging device at a predetermined period, said means for 
updating the data of said reference background picture contains means for calculating a median of said read pic- 

45 ture data composed of the predetermined number of frames at each pixel and operates to replace the data of said 
reference background picture saved in said storage means with said median value of each pixel as the new data of 
said reference background picture. 

8. An apparatus as claimed in claim 7, wherein said predetermined number of frames read from said imaging device 
so by said means for reading the data is smaller than the number of frames output by said imaging device at said pre- 
determined period. 

9. An apparatus as claimed in claim 7, wherein the period at which said data reading means reads frame data from 
said imaging device is longer than the frame period of output of said imaging device. 

55 

10. An apparatus as claimed in claim 7, wherein the data of said reference background picture and the data of said 
taken picture contain data about : luminance of plural pixels composing the picture. 

II. An apparatus as claimed in claim 7, wherein said means for sensing whether or not said object exists contains 
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means (1 30) for issuing an alarm if said object is sensed and said updating means operates to update the data of 
said reference background picture if said sensing means does not sense said object. 

1 2. An apparatus as claimed in claim 1 1 , wherein said means for reading the data contains first data reading means 
s for reading picture data composed of N1 frames and second data reading means for reading picture data com- 
posed of less N2 frames than said N1 if said object is not sensed, 

said means for calculating a median contains first median calculating means for calculating a median value 
of said read picture data composed of the N1 frames at each pixel and second median calculating means for cal- 
culating a median value of said read picture data composed of the N2 frames at each pixel, and 
io said updating means operates to replace the data of said reference background picture saved in said stor- 

age means with the median value of any one of said first and second median calculating means as the new data of 
said reference background picture. v - 

1 3. A method for sensing an object within an imaging field of an imaging device, comprising the steps of; 

15 

updating (143) data of a reference background picture within said imaging field, said data stored in storage 
means; 

reading (1 41 , 1 42) data of a picture taken by said imaging device; 

comparing (1 2) the data of said reference background picture stored in said storage means with said read pic- 
20 ture data at each pixel for calculating a difference value (en) of each pixel; 

comparing (15, 16) said difference value with a predetermined threshold value at each pixel and producing 
binarization data about the picture of said object based on the compared result; and 
: sensing (1 7) whether or not said object exists based on said binarization data. 

25 14. A method as claimed in claim 13, wherein when calculating said difference value, a process is executed to sense 
a polarity of said difference value, and at the step of producing said digital data, a process is executed to use a first 
threshold value (th + ) if the polarity is positive (1 5) or a second threshold value (th~) if the polarity is negative (1 6) . 

1 5. A method as claimed in claim 1 4, further including the step of comparing the data of said reference background pic- 
30 ture with the data of said taken picture at each block consisting of the predetermined number of pixels and calcu- 
lating a similarity (yn) of the data value between both of said data blocks based on the compared result, and in 
which said object is sensed on said similarity and said digital data. 

16. A method as claimed in claim 15, wherein said step of calculating the similarity is located after said step of creating 
35 the digital data and the similarity is calculated only for the pixels whose difference values depart from said threshold 

value. 

17. A method as claimed in claim 16, further including a step of comparing said calculated similarity with a predeter- 
mined reference value and at said step of sensing the object, it is determined that said object exists if said Similarity 

40 is higher than said reference valu6. 

1 8. A method as claimed in claim 1 7, wherein the data of said reference background picture and the data of said taken 
picture contain data about luminance of plural pixels composing the picture, said object similarity is a value for rep- 
resenting a similarity of distribution pattern of local luminance values of said picture, and at said step of sensing 

45 said object, the process is executed to .determine as existence of no object an area having pixels whose difference 
values have a positive polarity and whose similarities are higher than said reference value (48). 

19. A method as claimed in claim 17, wherein the data of said reference background picture and the data of said taken 
picture contain data about luminance of plural pixels composing the picture, the similarity of said object is a value 

so for representing the similarity of a distribution pattern of local luminance values of said picture, and at said step of 
sensing said object, a process is executed to determine as existence of no object an area Having pixels whose dif- 
ference values have a negative polarity and whose similarities are higher than said reference value (58). 

20. A method as claimed in claim 16, further including a step (36) of comparing said calculated similarity with a prede- 
55 termined reference value and steps (37, 38) of distinguishing an area having pixels whose similarities are higher 

than said reference value from an area having pixels whose similarities are lower than said reference value. 

21 . A method as claimed as claim 20, wherein said imaging device is an infrared imaging device, the data of said ref- 
erence background picture ^nd the data of said taken picture contain data about luminance of plural pixels com- 
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posing the picture, the similarity of said object is a value for representing the similarity of a distribution pattern of 
local luminance values of the picture, and at said step of distinguishing said areas from each other, a process is 
executed to determine as an area having a higher temperature than another area an area having pixels whose dif- 
ference values have a positive polarity and whose similarities are smaller than or equal to said reference value (68). 

5 . ■* 

22. A method as claimed in claim 20, wherein said imaging device is an infrared one, the data of said reference back- 
ground picture and the data of said taken picture contain data about luminance of plural pixels composing the pic- 
ture, the similarity of said object is a value for representing the similarity of a distribution pattern of local pixel values 
of said picture, and at said step for distinguishing said areas from each other, a process is executed to determine 

10 as an area whose temperature is made lower than the other areas an area haying pixels whose difference values 
have a negative polarity and whose similarities are smaller than or equal to said reference value (78). . 

23. A method as claimed in claim 13, wherein at said step of reading the data, said picture data composed of the pre- 
determined number of frames is read from said imaging device at a predetermined period, and said step of updat- 

75 ing the data of said reference background picture contains a step (304) of calculating a median (M) of said read 
picture data composed of the predetermined number of frames at each pixel and a step (305) of replacing the data 
of said reference background picture stored in said storage means with said median value of each pixel. 

24. A method as claimed in claim 23, wherein at said step of reading the data, said predetermined number of frames 
20 read from said imaging device is smaller than the number of frames output by said imaging device at said prede- 
termined period. 

A method as claimed in claim 23, wherein at said step of reading the data, the period at which the frame data is 
read from said imaging device is longer than the period of the frame output by said imaging device. 

A method as claimed in claim 23, wherein said step of sensing the object contains a step of issuing an alarm if said 
object is sensed and at said updating step, a process is executed to update the data of said reference picture if said 
object is not sensed. 

A method as claimed in claim 23, wherein said step of reading the data contains a first data reading step (603) of 
reading picture data composed of N1 frames and a second data reading step (610) of reading picture data com- 
posed of N2 frames that is less than said N1 if said object is not sensed (N1 and N2 are both a positive integer of 
2 or more), 

said step of calculating the median contains a first median calculating step (604) of calculating a median of 
said read picture data composed of the N1 frames at each pixel and a second median calculating step (61 1) of cal- 
culating a median of said read picture data composed of the N2 frames at each pixel, and 

at said step of updating the data, a process is executed to replace the data of said reference background 
picture stored in said storage means with a median value derived by any one of said first and second median cal- 
culating steps as the data of said reference background picture. 
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